Image processing system, apparatus, method and storage medium

ABSTRACT

An image processing system includes a first setting unit configured to set, in a learning image, a plurality of mutually different position coordinates belonging to a region of interest presenting a site of interest rendered in the learning image, an extracting unit configured to extract the region of interest by using the plurality of position coordinates, a calculating unit configured to calculate a feature value for determining an attribute of the site of interest from a plurality of extraction results of the region of interest corresponding to the plurality of position coordinates, and a constructing unit configured to construct an identifier for determining the attribute based on a plurality of the feature values corresponding to the plurality of extraction results of the region of interest and a correct answer value of the attribute of the site of interest rendered in the learning image.

BACKGROUND OF THE INVENTION Field of the Invention

The technology disclosed herein relates to an image processing system,an apparatus, a method and a storage medium.

Description of the Related Art

In recent years, many diagnoses and treatments have been performed inmedical fields which use medical images such as computer tomography (CT)images and nuclear magnetic resonance (MR) images. In a diagnosis usingsuch a medical image (image diagnosis), a doctor may find an abnormalshadow from a medical image for diagnosis, obtain an attribute of theabnormal shadow, and discriminate the abnormal shadow based on theattribute and clinical information that is obtained in advance.

For the purpose of aiding image diagnoses by a doctor, a computer aideddiagnosis or CAD apparatus has been developed which automatically infersand presents what an abnormal shadow in a medical image corresponds to.For example, an apparatus may be considered which calculates aprobability that an abnormal shadow (corresponding to a lung tuber) in achest CT image is a malignant tumor and a probability that it is abenign tumor and presents the results.

Japanese Patent Laid-Open No. 2016-7270 discloses a technology whichobtains a region corresponding to an abnormal shadow that is a subject(hereinafter, subject abnormal shadow) from a medical image and itsfeature value to discriminate whether the subject abnormal shadow isbenign or malignant. This technology identifies a region correspondingto a subject abnormal shadow (called region extraction) from a medicalimage based on positional information (seed point), which is first inputby a doctor, of the subject abnormal shadow. After that, a feature valueof the identified region is calculated, and whether the subject abnormalshadow is benign or malignant is inferred based on the obtained featurevalue.

SUMMARY OF THE INVENTION

An image processing system according to an aspect of the presentdisclosure includes a first setting unit configured to set, in alearning image, a plurality of mutually different position coordinatesbelonging to a region of interest presenting a site of interest renderedin the learning image, an extracting unit configured to extract theregion of interest by using the plurality of position coordinates, acalculating unit configured to calculate a feature value for determiningan attribute of the site of interest from a plurality of extractionresults of the region of interest corresponding to the plurality ofposition coordinates, a constructing unit configured to construct anidentifier for determining the attribute based on a plurality of thefeature values corresponding to the plurality of extraction results ofthe region of interest and a correct answer value of the attribute ofthe region of interest rendered in the learning image, a second settingunit configured to set, in an identification image, position coordinatesbelonging to the region of interest presenting the site of interestrendered in the identification image, and a determining unit configuredto determine an attribute of the site of interest rendered in theidentification image by using the identifier and the positioncoordinates set by the second setting unit.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a device configuration of an imageprocessing system according to a first embodiment.

FIG. 2 illustrates an example of a functional configuration of the imageprocessing system according to the first embodiment.

FIGS. 3A and 3B illustrate examples of procedures to be performed by theimage processing apparatus according to the first embodiment.

FIG. 4 illustrates an example of generation of reference pointinformation according to the first embodiment.

FIGS. 5A and 5B illustrate examples of generation of a region data groupaccording to the first embodiment.

FIG. 6 illustrates examples of generation of a feature value data groupand construction of an identifier according to the first embodiment.

FIG. 7 illustrates an example of a functional configuration of an imageprocessing system according to a second embodiment.

FIGS. 8A to 8C illustrate examples of procedures to be performed in theimage processing apparatus according to the second embodiment.

FIGS. 9A and 9B illustrate examples of construction of a reference pointgeneration model according to the second embodiment.

FIG. 10 illustrates an example of a functional configuration of an imageprocessing system according to a third embodiment.

FIGS. 11A to 11D illustrate examples of procedures to be performed by animage processing apparatus according to the third embodiment.

DESCRIPTION OF THE EMBODIMENTS

With reference to drawings, embodiments will be exemplarily described indetail. It should be noted that components in the embodiments are givenfor illustration purpose only, and the technical scope of the presentdisclosure is defined by the claims and is not limited by the followingindividual embodiments.

First Embodiment

Outline

An image processing apparatus according to a first embodiment performstwo processes of a learning process and an identification process. Inthe learning process, the image processing apparatus first obtains aplurality of position coordinates (reference points) belonging to a lungtuber to be processed (hereinafter, subject lung tuber) from a learningimage and obtains a plurality of extraction results (masked images) ofthe subject lung tuber based on their corresponding reference points.The lung tuber here corresponds to an example of a site of interest, andthe region of the lung tuber corresponds to an example of a region ofinterest. Next, the image processing apparatus calculates feature valuesof the subject lung tuber corresponding to the obtained masked imagesand registers them with learning data simultaneously with a correctanswer value of an attribute of the subject lung tuber to be obtained.The image processing apparatus constructs an identifier configured toobtain (or infer) an attribute of the subject lung tuber by using theplurality of the registered learning data pieces. Then in theidentification process, the image processing apparatus uses theidentifier constructed in the learning process to infer an attribute ofthe lung tuber in an identification image. Here, the term “attribute”may refer to a benign or malignant lung tuber or image findings of alung tuber, for example. Although a lung tuber on a CT image is to beprocessed in the following description, the application range of thisembodiment is not limited by a subject organ, a tumor, or the type of amodality. A device configuration, a functional configuration and aprocessing flow will specifically be described below.

Device Configuration

With reference to FIG. 1, an image processing system 190 including animage processing apparatus 100 according to the first embodiment of thepresent disclosure and apparatuses connected to the image processingapparatus 100 will be described in detail. The image processing system190 includes an imaging apparatus 110 configured to capture an image, adata server 120 configured to store the captured image, the imageprocessing apparatus 100 configured to perform an image process, adisplay unit 160 configured to display an obtained input image and animage process result, and an operating unit 170 usable by a user forinputting an instruction. The image processing apparatus 100 isconfigured to obtain an input image and perform an image process on aregion of interest shown on the input image. The input image maycorrespond to an image obtained by performing an image process forobtaining an optimum image for diagnosis on image data obtained by theimaging apparatus 110, for example. The input image according to thisembodiment is an image for learning (which will be called a learningimage) or an image for identification (which will be called anidentification image). These components will be described below. Theimage processing apparatus 100 may be a computer, for example, and mayperform image processing according to this embodiment. The imageprocessing apparatus 100 has a central processing unit (CPU) 11, a mainmemory 12, a magnetic disk 13, and a display memory 14. The CPU 11comprehensively controls operations of the components of the imageprocessing apparatus 100.

Under control of the CPU 11, the image processing apparatus 100 may alsocontrol operations performed by the imaging apparatus 110. The mainmemory 12 stores a control program to be executed by the CPU 11 andprovides a work area for execution of the program by the CPU 11. Themagnetic disk 13 is configured to store application software includingan operating system (OS), a device driver for a peripheral apparatus, aprogram for performing image processing according to this embodiment.The display memory 14 temporarily stores data to be displayed on thedisplay unit 160. The display unit 160 may be a liquid crystal monitorand is configured to display an image based on an output from thedisplay memory 14. The operating unit 170 may be a mouse or a keyboard,for example, and is usable by an operator for inputting positionalinformation or for inputting text. The display unit 160 may be a touchpanel monitor configured to receive an operation input, and theoperating unit 170 may be a stylus pen. These components are mutuallycommunicably connected via a common bus 18.

The imaging apparatus 110 may be a computed tomography (CT) apparatus, amagnetic resonance imaging (MRI) apparatus, or a digital radiography(DR) apparatus configured to capture a two-dimensional radiographicimage, for example. The imaging apparatus 110 transmits an obtainedimage to the data server 120. An imaging control unit, not illustrated,configured to control the imaging apparatus 110 may be included in theimage processing apparatus 100.

The data server 120 is configured to hold an image captured by theimaging apparatus 110. The data server 120 may be a picture archivingand communication system (PACS) server, for example. The imageprocessing apparatus 100 obtains an image from the data server 120 overa network such as a local area network (LAN).

Functional Configuration

Next, with reference to FIG. 2, a functional configuration of the imageprocessing system including the image processing apparatus 100 accordingto this embodiment will be described. The CPU 11 executes a programstored in the main memory 12 to implement functions of the componentsillustrated in FIG. 2. A subject that executes a program may be one ormore CPUs, and one or more main memories may be provided as the mainmemory for storing the program. Instead of or in addition to the CPU,another processor such as a graphics processing unit (GPU) may be used.In other words, at least one processor (hardware) may execute a programstored in at least one memory communicably connected to the processor sothat the functions of the components illustrated in FIG. 2 can beimplemented.

The image processing apparatus 100 has a learning unit 130, anidentifying unit 140 and a display control unit 1090 in its functionalconfiguration. The learning unit 130 further has a learning dataobtaining unit 1000, a reference point group generating unit 1010, aregion data group generating unit 1020, a feature value data groupgenerating unit 1030, and an identifier constructing unit 1040. Theidentifying unit 140 has an identification data obtaining unit 1005, areference point obtaining unit 1015, a region extracting unit 1025, afeature value calculating unit 1035, and an attribute inferring unit1050. The image processing apparatus 100 is communicably connected tothe data server 120 and the display unit 160.

The learning unit 130 is configured to construct an identifier to beused for inferring an attribute of a subject lung tuber.

The learning data obtaining unit 1000 is configured to obtain learningdata for constructing an identifier according to this embodiment. Thelearning data may include a learning image obtained from the data server120 for undergoing an image process and correct answer data of anattribute of the subject lung tuber rendered on the learning imageobtained from the operating unit 170.

The reference point group generating unit 1010 is configured to generateinformation (hereinafter, reference point information) regarding aplurality of points (hereinafter, reference points) belonging to aregion corresponding to the subject lung tuber from the learning imageobtained by the learning data obtaining unit 1000.

The region data group generating unit 1020 is configured to generate aplurality of region data pieces (masked images) representing a subjectlung tuber region based on the learning image obtained by the learningdata obtaining unit 1000 and the plurality of reference pointinformation pieces obtained by the reference point group generating unit1010. In other words, the region data group generating unit 1020generates region data for each of the plurality of reference pointinformation pieces to generate a plurality of region data representingthe subject lung tuber region. The region data group generating unit1020 can generate a masked image by using an arbitrary segmentationmethod such as a region extension method. The region data groupgenerating unit 1020 can perform a region extension method based onimage feature values such as pixel values (concentration values) ofpixels corresponding to the reference point information to generate amasked image.

The feature value data group generating unit 1030 is configured toobtain a feature value data group representing characteristics of thesubject lung tuber by using the learning image obtained by the learningdata obtaining unit 1000 and the plurality of region data piecesobtained by the region data group generating unit 1020.

The identifier constructing unit 1040 is configured to construct anidentifier for inferring an attribute based on the correct answer dataof the attribute of the subject lung tuber obtained by the learning dataobtaining unit 1000 and the feature value data group obtained by thefeature value data group generating unit 1030, which are registeredtherewith as learning data.

The identifying unit 140 is configured to infer an attribute of thesubject lung tuber rendered on the identification image by using asubject image for identification (hereinafter, identification image)obtained by the identification data obtaining unit 1005.

The identification data obtaining unit 1005 is configured to obtain theidentification image included in a subject of the identificationaccording to this embodiment from the data server 120.

The reference point obtaining unit 1015 is configured to obtaininformation regarding a reference point belonging to a regioncorresponding to the subject lung tuber rendered on the identificationimage obtained by the identification data obtaining unit 1005.

The region extracting unit 1025 is configured to obtain a masked image(extraction result) representing a region corresponding to the subjectlung tuber based on the identification image obtained by theidentification data obtaining unit 1005 and the reference pointinformation regarding the subject lung tuber obtained by the referencepoint obtaining unit 1015. The region extracting unit 1025 can generatea masked image by using an arbitrary segmentation method such as aregion extension method. The region extracting unit 1025 can perform aregion extension method based on image feature values such as pixelvalues (concentration values) of pixels corresponding to the referencepoint information to generate a masked image.

The feature value calculating unit 1035 is configured to calculate afeature value of the subject lung tuber by using the identificationimage obtained by the identification data obtaining unit 1005 and anextraction result corresponding to the subject lung tuber obtained bythe region extracting unit 1025.

The attribute inferring unit 1050 is configured to infer an attribute ofthe subject lung tuber based on the feature value, which is input to theidentifier constructed by the identifier constructing unit 1040 andobtained by the feature value calculating unit 1035.

The display control unit 1090 is configured to output informationregarding a subject obtained through processes performed by theidentifying unit 140 to the display unit 160 and to cause the displayunit 160 to display results of the processes.

At least a part of the components of the image processing apparatus 100may be implemented as an independent apparatus. The image processingapparatus 100 may be a work station. The functions of the components maybe implemented as software which runs on a computer, and the softwareimplementing the functions of the component may run on a server via anetwork such as a cloud. The following embodiments assume that thecomponents are implemented by software which runs on a computerinstalled in a local environment.

Processing Flow

Next, image processes according to the first embodiment of the presentdisclosure will be described. FIGS. 3A and 3B illustrate procedures ofprocesses to be executed by the image processing apparatus 100 accordingto this embodiment. This embodiment is implemented by a program executedby the CPU 11, where the program is stored in the main memory 12 andimplements functions of the components. According to this embodiment, aCT image is to be processed. The CT image is obtained as athree-dimensional gray image. According to this embodiment, a lung tuberis an example of a processing subject included in a subject image.

First, processing (steps S1100 to S1140) to be executed in a learningprocess by the image processing apparatus 100 according to the firstembodiment will be described. This process is executed in a stage wherethe image processing apparatus 100 according to this embodiment isconstructed. In the learning process, after the image processingapparatus 100 according to the first embodiment applies processing fromsteps S1100 to S1130 on a plurality of learning data pieces (details ofwhich will be described below), all of the results therefrom are used toperform the processing in step S1140. For omission of any repetitivedescriptions, a case will be described in which the processing from stepS1110 to step S1130 is applied to one learning data.

S1100

In step S1100, the learning data obtaining unit 1000 obtains a learningdata piece A to construct an identifier configured to identify anattribute of a subject lung tuber. The learning data A is a set of aprocessing subject image (learning image) I_(A)(x,y,z) and correctanswer data piece D_(A) of an attribute of a subject lung tuber renderedin the learning image.

Here, the learning image according to this embodiment includes aplurality of pixels at positions located based on three-dimensionalorthogonal coordinates (x,y,z). A pixel size that is one attribute ofthe image is defined for each of directions of three-axis coordinates.According to this embodiment, the pixel sizes in x, y, z directions arereferred to as r_size_x, r_size_y, and r_size_z, respectively, and arelarger than 1.0 mm though the pixel sizes are not limited to the values.The pixel values of the subject image are determined for each ofthree-dimensional coordinates (x,y,z). Therefore, the subject image canbe regarded as data defined by a function I_(A)(x,y,z) having athree-dimensional coordinate value as an argument.

First, the learning data obtaining unit 1000 obtains a CT image that isa processing subject image from the data server 120 and stores it in themain memory 12 in the image processing apparatus 100. As anotherexample, the learning data obtaining unit 1000 obtains an image datacaptured by the imaging apparatus 110 via a communication unit, performsan image process for obtaining an image suitable for diagnosis andobtains the result as a learning image according to this embodiment. Ina case where the imaging apparatus 110 is a CT apparatus, for example, aCT image data piece including a pixel value called a HU (HounsfieldUnit) value is obtained from the imaging apparatus 110.

Next, the learning data obtaining unit 1000 obtains a correct answerdata piece D_(A) for an attribute of a subject lung tuber rendered inthe obtained learning image. The correct answer data piece for anattribute can be a correct answer value for the attribute of the subjectlung tuber rendered at I(x,y,z) which is input by an operator (doctor)through the operating unit 170, for example.

S1110

In step S1110, the reference point group generating unit 1010 generates,as a reference point, n points P_(Ai)(x_(Ai), y_(Ai), z_(Ai)) (i=1, 2, .. . , n) belonging to the subject lung tuber in the learning imageI_(A)(x,y,z). The n points P_(Ai) may be at different positions fromeach other, for example. Here, a set having P_(Ai) as its component isassumed as a reference point group P_(A)={P_(Ai)|i=1, 2, . . . , n}. Anexample of the generation of a reference point group from a learningimage will be described.

First, with reference to axial, sagittal and coronal tomographic images,for example, of the learning image I_(A)(x,y,z) displayed on the displayunit 160, an operator may select one pixel included in the subject lungtuber through the operating unit 170. The reference point groupgenerating unit 1010 then obtains the pixel obtained through theoperation input as an initial reference point P_(A1). Next, thereference point group generating unit 1010 selects another referencepoint with reference to P_(A1) and obtains a reference point groupP_(Ai)(i=2, 3, . . . , n). This processing may be executed on a localimage region V_(A)(x,y,z) including P_(A1), for example. The local imageregion V_(A)(x,y,z) is cropped from the learning image I(x,y,z) aboutP_(A1) based on a predetermined size. Alternatively, an operator canmanually crop the local image region V_(A)(x,y,z) from I_(A)(x,y,z)through the operating unit 170. Furthermore, a technology based on ascale space such as a Laplacian of Gaussian (LoG) kernel is used toroughly estimate the size of the subject lung tuber, and the local imageregion V_(A)(x,y,z) can be set based on the estimated size. FIG. 4illustrates examples of a local image 510 including a subject lung tuber550 and an initial reference point 600 which are obtained from thelearning image 500.

The reference point group P_(A) is obtained by applying an image processunit 1 to the local image region V_(A)(x,y,z). The image process unit 1may be a process for searching a pixel having a feature similar to thatof P_(A1) from V_(A)(x,y,z), for example. Here, the term “feature”refers to a concentration value, V_(A)(x,y,z) or the like. The imageprocess unit 1 may be a process for searching a pixel having a highpossibility (likelihood) of belonging to the subject lung tuber fromV_(A)(x,y,z) based on advance knowledge. As an example of this case, ablobness structure enhancement filter based on an Eigen value of aHessian matrix is applied to V_(A)(x,y,z), and a pixel having a highfilter output level is selected as a reference point to generate areference point set P. Instead of the blobness structure enhancementfilter, an LoG kernel or a Scale-Invariant Feature Transform (SIFT)feature point detection process may be used as the image process unit 1.

S1120

In step S1120, the region data group generating unit 1020 generates aregion data group of the subject lung tuber from the learning imageI_(A)(x,y,z) through a region extraction process. The region data groupgenerating unit 1020 performs the region extraction process with respectto the reference points by using the learning image I_(A)(x,y,z)obtained in step S1100 and the reference point group P_(A) and the localimage region V_(A)(x,y,z) obtained in step S1110. Through this process,region data (masked image) of the lung tuber can be obtained which has aone-to-one correspondence relation with each of the reference points.With reference to FIGS. 5A and 5B, a region extraction process to beperformed in a case where four (n=4) reference points are obtained instep S1110 will be described below. The number of reference points isonly required to be plural and is not limited to the value above.

As illustrated in FIG. 5A, the region data group generating unit 1020performs a region extraction process on the subject lung tuber 550 basedon a reference point 600, a reference point 601, a reference point 602,and a reference point 603, and thus obtains region data corresponding tothe reference points. The region extraction process may be performed onI_(A)(x,y,z) but may be performed on a local image region V_(A)(x,y,z).According to this embodiment, the local image region 510 is used toperform the region extraction process. The region extraction process isperformed by a region extraction algorithm such as a region extensionmethod, a Level-set method or a Graph-cut method by defining a referencepoint as a foreground region (seed) of the subject lung tuber region.The region extraction process may use one predetermined regionextraction algorithm for all of subject lung tubers, or the regionextraction algorithm may be changed in accordance with the shape orproperty of the subject lung tuber. For example, machine learning or aGraph-cut method may be used for a lung tuber with ground glass opacity(GGO) while a Level-set method may be used for a lung tuber having anirregular shape and a small cubic content. The extraction algorithm maybe changed based on a characteristic of each of different referencepoints in one subject lung tuber.

An operator may visually evaluate accuracy of the extraction from theplurality of obtained region data pieces of the subject lung tuber andmay delete a region data piece having a low accuracy. Then, the regiondata group finally obtained from the learning image I_(A)(x,y,z) isdefined as R_(A)={R_(Aj)|=1, 2, . . . , m} (m<n). FIG. 5B illustratesexamples of a region data piece 710, a region data piece 711, a regiondata piece 712, and a region data piece 713 obtained based on thereference point 600, the reference point 601, the reference point 602,and the reference point 603, respectively.

S1130

In step S1130, the feature value data group generating unit 1030 obtainsa feature value data group of the subject lung tuber. The feature valuedata group generating unit 1030 calculates a feature value of thesubject lung tuber based on the learning image I_(A)(x,y,z) obtained instep S1100, the local image region V_(A)(x,y,z) obtained in step S1110,and the region data group R_(A) of the subject lung tuber obtained instep S1120.

The calculation of a feature value of the subject lung tuber starts froma region data piece R_(A)j in the region data group R_(A) of the subjectlung tuber obtained in step S1120. For example, in a case where afeature value is to be calculated by using the region data piece R_(A1),the feature value data group generating unit 1030 calculates a shapefeature value region represented by the data piece R_(A1) or a texturefeature value of a region overlapping R_(A1) or a surrounding region ofthe region R_(A1) in I_(A)(x,y,z). The feature value may be a publiclyknown general image feature value pr a feature value calculated by acalculation method based on an attribute of a subject lung tuber to beobtained. For example, in a case where an attribute to be inferred bythe image processing apparatus 100 is image findings of the subject lungtuber, a special feature value may be calculated based on a clinicalimage findings item frequently used by a doctor.

A feature value data piece F_(A1) is calculated based on the region datapiece R_(A1). The feature value data group obtained after the featurevalue calculations are performed based on all region data pieces R_(A)is referred to as F_(A)={F_(Aj)|=1, 2, . . . , m}.

In a case where a correct answer region R_(G) of the subject lung tuberis prepared in advance in the learning image, the feature value datagroup generating unit 1030 calculates a feature value of the subjectlung tuber based on the correct answer region R_(G) to obtain a featurevalue F_(AG). The feature value data group generating unit 1030 maycalculate a matching degree between F_(AG) and each feature value ofF_(A)={F_(Aj)|=1, 2, . . . , m} and may delete a feature value with alower matching degree from F_(A). Here, the matching degree calculationmay be based on a Mahalanobis distance in a feature space, for example.This processing excludes a feature value data piece calculated from anextraction result with lower accuracy from training data for identifierconstruction to prevent reduction of accuracy of an identifier to beconstructed in a next step.

S1140

In step S1140, the identifier constructing unit 1040 constructs anidentifier for inferring an attribute of the lung tuber. First, theidentifier constructing unit 1040 registers a set of the correct answerdata D_(A) of the attribute obtained in step S1100 and the feature valuegroup data F_(A) obtained in step S1130 as training data with the mainmemory 12 in the image processing apparatus 100. In other words, theidentifier constructing unit 1040 registers the correct answer dataD_(A) and the feature value group data F_(A) with the main memory 12 inassociation. Here, a plurality of training data pieces (training datagroup) is registered by repeating the processing from step S1100 to stepS1130 as required.

With reference to FIG. 6, the registration of a training data group willbe described. It is assumed here that feature values 810, 811, 812, and813 correspond to the region data pieces 710, 711, 712, and 713 obtainedfrom the four reference points in the subject lung tuber 550. By settingthe feature values and the correct answer 900 of the attribute, trainingdata pieces 910, 911, 912 and 913 can be obtained. The same processingis performed on other subject lung tubers, and all training data piecesobtained therefrom are registered as a training data group.

After the training data group is registered, the identifier constructingunit 1040 constructs an identifier by using the registered training datagroup. The identifier applies a publicly known technology such as RandomForest (RF), SupportVector Machine (SVM), or Neural Network. Thetraining data group is input to the identifier and an optimum parameterfor obtaining an attribute from the feature value is searched toconstruct an identifier by which the attribute of the lung tuber can beidentified. The constructed identifier is referred to as C_(pro).

Up to this point, the processing for constructing an identifier C_(pro)by the image processing apparatus 100 according to this embodiment byusing learning data (or processing of the learning process) has beendescribed. Next, processing for inferring an attribute of a lung tuberrendered in an unknown image (identification image) by using theconstructed C_(pro) (processing of an identification process in stepsS1150 to S1200) will be described.

S1150

In step S1150, the identification data obtaining unit 1005 obtains anidentification image for identifying an attribute of a lung tuber. Inother words, the identification data obtaining unit 1005 obtains anidentification image from the data server 120 and stores it in the mainmemory 12 in the image processing apparatus 100.

Because the identification image acquisition is performed by the sameprocessing as the acquisition of learning data from the data server 120by the learning data obtaining unit 1000 in step S1100, any repetitivedescriptions will be omitted. The obtained identification image isreferred to as I_(rec)(i,j,k).

S1160

In step S1160, the reference point obtaining unit 1015 obtains areference point of a lung tuber to be identified (identification subjectlung tuber). The reference point obtaining unit 1015 obtains a pixelP_(seed_rec) belonging to a region of the identification subject lungtuber rendered in the identification image I_(rec)(i,j,k) obtained instep S1150.

The P_(seed_rec) can be manually selected with reference to atomographic image of the identification image I_(rec)(i,j,k) displayedon the display unit 160 by an operator, like the initial reference pointP_(A1) in step S1100. The initial reference point P_(A1) may beautomatically detected from I_(rec)(i,j,k) by using a reference pointdetection scheme.

S1170

In step S1170, the region extracting unit 1025 extracts a region of theidentification subject lung tuber from the identification image. Theregion extracting unit 1025 extracts the region of the identificationsubject lung tuber at I_(rec)(i,j,k) and obtains its extraction resultbased on the identification image I_(rec)(i,j,k) obtained in step S1150and the reference point P_(seed _rec) obtained in step S1160.

Because the extraction of the region of the identification subject lungtuber by the region extracting unit 1025 is the same as the processingfor obtaining region data pieces corresponding to the reference pointsby the region data group generating unit 1020 in step S1120, anyrepetitive descriptions will be omitted. The extraction result (maskedimage) of the identification subject lung tuber obtained from theidentification image is referred to as R_(rec)(i,j,k).

S1180

In step S1180, the feature value calculating unit 1035 calculates afeature value of the identification subject lung tuber. The featurevalue calculating unit 1035 calculates a feature value of theidentification subject lung tuber based on the identification imageI_(rec)(i,j,k) obtained in step S1150 and extraction resultR_(rec)(i,j,k) obtained in step S1170.

Because the calculation of a feature value of the identification subjectlung tuber is the same as the processing for calculating feature valuescorresponding to region data pieces of the region data group R_(A) bythe feature value data group generating unit 1030 in step S1130, anyrepetitive descriptions will be omitted. The obtained feature value ofthe identification subject lung tuber is referred to as F_(rec).

S1190

In step S1190, the attribute inferring unit 1050 infers an attribute ofthe identification subject lung tuber. The attribute inferring unit 1050inputs the feature value F_(rec) obtained in step S1180 to theidentifier C_(pro) constructed in step S1140 to obtain an attribute ofthe subject lung tuber.

The feature value F_(rec) input to the identifier C_(pro) is projectedto an identification space generated with feature values of trainingdata used for constructing the C_(pro), and a class of the attribute towhich the feature value F_(rec) belongs in the identification space isobtained so that the attribute of the subject lung tuber can beinferred.

S1200

In step S1200, the display control unit 1090 controls to display theprocessing result on the display unit 160. The display control unit 1090causes the display unit 160 to display at least one of the extractionresult of the identification subject lung tuber, the feature valueinformation, and the attribute inference result. In this case, thedisplay control unit 1090 transmits the extraction result, the featurevalue information and the information regarding the attribute to thedisplay unit 160 connected to the image processing apparatus 100 andcontrols to display them on the display unit 160.

The display control unit 1090 superimposes these information pieces onthe identification image I_(rec)(i,j,k) that is the input image fordisplay on the display unit 160. In this case, the display control unit1090 may generate three-dimensional image information having thoseinformation pieces thereon by rendering for display on the display unit160. The display control unit 1090 may generate a predeterminedcross-sectional image of the superimposed three-dimensional image fordisplay on the display unit 160.

Effects of the image processing apparatus according to the firstembodiment will be described. The image processing apparatus accordingto this embodiment can solve a problem present in a publicly knowntechnology relating to discrimination of an attribute of an abnormalshadow. The problem is that a conventional discrimination technologywhich receives different positional information pieces regarding anidentical lung tuber region provides different region extraction resultsand different feature value calculation results and, as a result,different attribute discrimination results may sometimes be obtained. Inother words, the problem is about reproducibility. The reproducibilityproblem may easily occur in a case where different operators performattribute discrimination processes on one lung tuber region, forexample. This is because different positional information pieces areinput when different operators input positional information pieces forone lung tuber region. This problem may occur when one operator inputsthe information pieces. Even when one operator inputs positionalinformation pieces at different times for one lung tuber region, theinput positional information pieces may differ.

On the other hand, in the image processing apparatus according to thisembodiment, based on a plurality of positional information pieces forone learning image, an abnormal shadow region is extracted, and featurevalues are calculated. Then, those results and a correct answer value ofan attribute of the abnormal shadow in the learning image are used aslearning data to construct an identifier for inferring the attribute.Thus constructed identifier even having received different positionalinformation pieces input by an operator or operators can absorbdifferences occurring in the extraction and the feature valuecalculation so that an attribute inference result with highreproducibility can be obtained.

Therefore, the image processing apparatus according to the firstembodiment of the present disclosure has an effect that an attribute ofan abnormal shadow can be inferred with high reproducibility even whenan operator or operators inputs or input different positionalinformation pieces regarding the abnormal shadow.

Second Embodiment

Outline

Next, with reference to drawings, an example of a second embodiment willbe described in detail. Any repetitive descriptions on like components,functions, and operations in the first and second embodiments will beomitted, and differences from the first embodiment will mainly bedescribed.

In the image processing apparatus according to the second embodiment, amodel generating unit 150 in step S2000 constructs a reference pointgeneration model. In step S2110, a reference point group generating unit1018 generates reference points of a subject lung tuber rendered in alearning image by using the reference point generation model. Becausethe reference point generation model is constructed from a referencepoint actually input by an operator, there is a high probability thatthe reference point group generated based thereon will be set in a realscene. Thus, a more appropriate attribute identifier can be constructed.Functional configuration and a processing flow will be specificallydescribed below.

Functional Configuration

With reference to FIG. 7, a functional configuration of an imageprocessing apparatus 200 according to the second embodiment of thepresent disclosure will be described. Like numbers to the numbers inFIG. 2 refer to like components having like functions in the first andsecond embodiments, and any repetitive descriptions will be omitted. Asillustrated in FIG. 7, the image processing apparatus 200 according tothis embodiment includes, in a functional configuration, the modelgenerating unit 150, a learning unit 133, an identifying unit 140, and adisplay control unit 1090. The model generating unit 150 may beimplemented by a processor, like the first embodiment. The learning unit133 further has a learning data obtaining unit 1000, a reference pointgroup generating unit 1018, a region data group generating unit 1020, afeature value data generating unit 1030, and an identifier constructingunit 1040. The identifying unit 140 has an identification data obtainingunit 1005, a reference point obtaining unit 1015, a region extractingunit 1025, a feature value calculating unit 1035, and an attributeinferring unit 1050. The image processing apparatus 200 is connected toa data server 120 and a display unit 160.

The model generating unit 150 is configured to construct a referencepoint generation model that is a probability model representing apossibility (likelihood) that positions (pixels) in an image space areset as reference points. The reference point group generating unit 1018is configured to generate a plurality of reference points from thelearning image based on the reference point generation model.

Processing Flow

Next, image processing according to the second embodiment will bedescribed. FIGS. 8A, 8B, and 8C illustrate procedures of a modelconstruction process, a learning process, and an identification processto be executed by the image processing apparatus 200 according to thisembodiment. This embodiment is implemented by a program executed by theCPU 11 where the program is stored in the main memory 12 and implementsfunctions of the components.

Because the processing in step S2100 and step S2120 to step S2200 is thesame as the processing in step S1100 and step S1120 to step S1200illustrated in FIG. 3, any repetitive detail description will beomitted.

S2000

In step S2000, the model generating unit 150 constructs a referencepoint generation model. The model generating unit 150 learns by using aplurality of input images (training images for the reference pointgeneration model construction) and information regarding referencepoints (hereinafter, regarding point information) belonging to a subjectlung tuber rendered in the images from the data server 120 andconstructs a reference point generation model. Here, the reference pointinformation may be input in advance (or in the past) or may be input byan operator through the operating unit 170.

With FIGS. 9A and 9B, the reference point generation model constructionwill be described. The reference point generation model is a probabilitymodel representing possibilities (likelihoods) that pixels aredesignated as reference points in an image space in the model. Theconstruction of a reference point generation model is started withprojection of reference point information regarding a plurality of inputtraining images 310 to 312 and reference points 400, 410, and 420 forreference point generation model construction including the subject lungtubers 350 to 352 to a normalized image space 530 as illustrated in FIG.9A. Here, the image to be projected may be an input training image ormay be a partial image of an input training image including a subjectlung tuber region. According to this embodiment, the image to beprojected is a partial image B_(A)(i,j,k) cropped in a bounding box forthe subject lung tuber region. One or more reference points are to bedesignated in each of the images to be projected. The reference point orpoints may be all designated by one operator or may be designated by aplurality of operators.

When the partial image B_(A)(i,j,k) and the reference point informationare projected to the normalized image space 530, the number of referencepoints projected to pixel positions in the normalized image space 530 ishandled as a parameter representing a likelihood of generation of areference point at each of the pixel positions.

The model generating unit 150 performs the projecting and the likelihoodcalculating processing on all of the input training images for referencepoint generation model construction to generate a model representingprobabilities that pixels in the normalized image space can be referencepoints, as illustrated in FIG. 9B, that is, to generate a referencepoint generation model 540.

S2110

In step S2110, the reference point group generating unit 1018 generatesa reference point set (reference point group). The reference point groupgenerating unit 1018 uses the reference point generation modelconstructed in step S2000 and the learning image obtained in S2100 toobtain a plurality of points belonging to the subject lung tuberrendered in the learning image as a reference point set (reference pointgroup).

Processing will be described which generates a reference point groupfrom a learning image by using the reference point group generationmodel. First, the reference point group generating unit 1018 projectsthe learning image to a normalized space of the reference pointgeneration model. Here, like the construction of a reference pointgeneration model, the reference point group generating unit 1018projects a partial image cropped in a bounding box for a subject lungtuber from the learning image. The reference point group generating unit1018 selects pixels as a reference point group P′_(A) on the partialimage of the learning image projected to upper n pixel positions indecreasing order of the generation likelihoods in the reference pointgroup generation model.

After that, the processing from step S2120 to step S2140 is sequentiallyperformed so that an identifier C_(pro) for identifying an attribute ofthe lung tuber can be constructed. The processing of the identificationprocess from step S2120 to step S2200 is performed in the same manner asthe processing from step S1150 to step S1200 so that an attribute of thesubject lung tuber for identification can be inferred.

According to the second embodiment as described above, a reference pointgeneration model is constructed based on a reference point input by anoperator and is used to generate a reference point group of the subjectlung tuber from a learning image. Thus, the reference point group can begenerated at a position with a high possibility that an operator inputsin a real scene so that an identifier more suitable for the scene can beconstructed. The constructed identifier is advantageously used to moreproperly aid a differential diagnosis.

Third Embodiment

Outline

Next, with reference to drawings, an example of a third embodiment willbe described in detail. Any repetitive descriptions on like components,functions, and operations in the first and second embodiments will beomitted, and a difference from the first and second embodiments willmainly be described.

In an image processing apparatus according to the third embodiment, thestorage unit 1060 in step S3300 of the identification process storesinformation such as the reference point information input by an operatorin step S3160. In step S3005, the model generating unit 155 performsrelearning (additional learning) by adding newly stored reference pointinformation to the reference point model so that the reference pointgeneration model can be updated. Through the additional learningprocess, the reference point generation model is updated to more fit toan actual custom of the operator. Functional components and a processingflow will be specifically described below.

(Functional Configuration)

With reference to FIG. 10, functional components included in an imageprocessing apparatus 300 according to the third embodiment of thepresent disclosure will be described. Like numbers to the numbers inFIG. 7 refer to like components having like functions in the second andthird embodiments, and any repetitive descriptions will be omitted. Asillustrated in FIG. 10, the image processing apparatus 300 according tothis embodiment includes, in a functional component, a model generatingunit 155, a learning unit 133, an identifying unit 145, and a displaycontrol unit 1090. The model generating unit 155 is also implemented bya processor, like the first embodiment. The learning unit 133 furtherincludes a learning data obtaining unit 1000, a reference point groupgenerating unit 1018, a region data group generating unit 1020, afeature value data generating unit 1030, and an identifier constructingunit 1040. The identifying unit 145 includes an identification dataobtaining unit 1005, a reference point obtaining unit 1015, a regionextracting unit 1025, a feature value calculating unit 1035, anattribute inferring unit 1050, and a storage unit 1060. The imageprocessing apparatus 300 is connected to a data server 120 and a displayunit 160. The storage unit 1060 is configured to store, in the mainmemory 12, information obtained in the reference point obtaining unit1015, the region extracting unit 1025, the feature value calculatingunit 1035, and the attribute inferring unit 1050. The model generatingunit 155 updates an existing reference point generation model byobtaining reference point information newly stored in the main memory 12and performing additional learning. Through the update, informationpieces input by an identical operator are gradually incorporated intothe reference point generation model so that a model that fits tocustoms of the operator can be constructed.

Processing Flow

Next, image processing according to the third embodiment will bedescribed. FIGS. 11A, 11B, 11C and 11D illustrate procedures of a modelconstruction process, a learning process, an identification process anda and model update process to be executed by the image processingapparatus 300 according to this embodiment. This embodiment isimplemented by a program executed by the CPU 11 where the program isstored in the main memory 12 and implements functions of the components.

Because the processing excluding processing in step S3300 in the modelconstruction process, the learning process and the identificationprocess is the same as the processing in the construction process, thelearning process, and the identification process according to the secondembodiment illustrated in FIGS. 8A to 8C, any repetitive detaildescriptions will be omitted.

S3300

In step S3300, the storage unit 1060 stores the reference pointinformation obtained in step S3160 in the main memory 12. At the sametime, the storage unit 1060 may further store in the main memory 12 theregion data obtained in step S3170, the feature value informationobtained in step S3180, and the attribute information obtained in stepS3190, for example.

S3005

In step S3005, the model generating unit 155 updates the reference pointgeneration model. The model generating unit 155 obtains anidentification image stored in the main memory 12 and reference pointinformation input by an operator. By using the obtained identificationimage and the reference point information, additional learning of thereference point generation model constructed in step S3000 is performed.The additional learning projects the identification image and thereference point information to a normalized image space of the existingreference point generation model and recalculates a reference pointlikelihood of each of pixels in the normalized space, like learning inthe model construction performed in step S2000. Through the projectingand likelihood recalculation processing, the reference point generationmodel is updated.

When the reference point model is updated, processing of the learningprocess from step S3100 to step S3140 is executed again to update theidentifier C_(pro). Furthermore, when an operator uses the C_(pro) toperform processing for identifying an attribute of a subject lung tuberfrom a new identification image, the identification process in FIG. 11C,the model update process in FIG. 11D, and the learning process in FIG.11B are sequentially repeated.

Thus, according to the third embodiment, one operator performs theattribute identification process on a subject lung tuber based on manyidentification images so that the reference point model and theidentifier C_(pro) can be updated to fit to customs of the operator forhigher accuracy of attribute identification.

Other Embodiments

The image processing apparatus and image processing system according tothe aforementioned embodiments may be implemented as a single apparatus,or apparatuses including a plurality of information processing devicesmay be communicably connected to execute the processes as describedabove, which are both comprehended in embodiments of the presentdisclosure. A common server apparatus or server group may execute theprocesses as described above. In this case, the common server apparatuscorresponds to an image processing apparatus according to an embodiment,and the server group corresponds to an image processing system accordingto an embodiment. A plurality of apparatuses included in the imageprocessing apparatus and the image processing system may communicate ata predetermined communication rate and may not necessarily exist withinan identical facility or an identical country.

Having described the examples of the embodiments in detail, the presentdisclosure can have modes as a system, an apparatus, a method, a programor a recording medium (storage medium), for example. More specifically,the present disclosure is applicable to a system including a pluralityof apparatuses (such as a host computer, an interface device, an imagingapparatus, and a Web application) or is applicable to an apparatusincluding one device.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

Having described the embodiments of the present disclosure in detail,the present disclosure is not limited to such specific embodiments, andvarious changes, modifications and alterations can be made theretowithout departing from the spirit and scope of the claimed presentdisclosure.

A configuration including a combination of the aforementionedembodiments is also comprehended in embodiments of the presentdisclosure.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-154460 filed Aug. 9, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing system comprising: a memorythat stores a program; and a processor that executes the program storedin the memory to function as units including: a first setting unitconfigured to set, in a learning image, a plurality of mutuallydifferent position coordinates belonging to a region of interestpresenting a site of interest rendered in the learning image; anextracting unit configured to extract the region of interest by usingeach of the plurality of position coordinates; a calculating unitconfigured to calculate a feature value for determining an attribute ofthe site of interest from each of a plurality of extraction results ofthe region of interest corresponding to each of the plurality ofposition coordinates; a constructing unit configured to construct anidentifier for determining the attribute based on training dataincluding a set of each of a plurality of the feature valuescorresponding to each of the plurality of extraction results of theregion of interest and a correct answer value of the attribute of thesite of interest rendered in the learning image; a second setting unitconfigured to set, in an identification image, position coordinatesbelonging to the region of interest presenting the site of interestrendered in the identification image; and a determining unit configuredto determine an attribute of the site of interest rendered in theidentification image by using the identifier and the positioncoordinates set by the second setting unit.
 2. The image processingsystem according to claim 1, wherein the first setting unit sets, in thelearning image, the position coordinates based on a feature of theregion of interest rendered in the learning image.
 3. The imageprocessing system according to claim 2, wherein the feature is aconcentration value of the region of interest rendered in the learningimage.
 4. The image processing system according to claim 1, wherein thefirst setting unit sets, in the learning image, the position coordinatesbased on position coordinates set in the past.
 5. The image processingsystem according to claim 4, further comprising: a generating unitconfigured to generate a model based on the position coordinates set inthe past, wherein the first setting unit sets, in the learning image,the position coordinates based on the model.
 6. The image processingsystem according to claim 5, wherein the model is a model representingprobabilities that pixels in an image are designated as positioncoordinates.
 7. The image processing system according to claim 5,wherein the model is updated by using positional information set for theidentification image.
 8. An apparatus comprising: a memory that stores aprogram; and a processor that executes the program stored in the memoryto function as units including: a setting unit configured to set, in alearning image, a plurality of position coordinates belonging to aregion of interest presenting a site of interest rendered in thelearning image; an extracting unit configured to extract the region ofinterest by using each of the plurality of position coordinates; acalculating unit configured to calculate a feature value for determiningan attribute of the site of interest from each of a plurality ofextraction results of the region of interest corresponding to each ofthe plurality of position coordinates; and a constructing unitconfigured to construct an identifier for determining the attributebased on training data including a set of each of a plurality of thefeature values corresponding to each of the plurality of extractionresults of the region of interest and a correct answer value of theattribute of the site of interest.
 9. A method comprising: setting, in alearning image, a plurality of position coordinates belonging to aregion of interest presenting a site of interest rendered in thelearning image; extracting the region of interest by using each of theplurality of position coordinates; calculating a feature value fordetermining an attribute of the site of interest from each of aplurality of extraction results of the region of interest correspondingto each of the plurality of position coordinates; and constructing anidentifier for determining the attribute based on training dataincluding a set of each of a plurality of the feature valuescorresponding to the plurality of extraction results of the region ofinterest and a correct answer value of the attribute of the site ofinterest.
 10. A non-transitory computer-readable storage medium storinga program causing a computer to execute the method according to claim 9.